System And Method For Capturing And Projecting Images, And Use Of The System

ABSTRACT

The invention relates to a system and a method for capturing and projecting images for use in an integrated studio including a real location and panels which render these premises partially virtual, using images generated from outside to create an image that is part real and part virtual by showing the images.

FIELD OF THE INVENTION

The invention relates to the field of television production, morespecifically to systems and methods of capturing and projecting images,and using them in TV Studio, and extending the images through virtualprojections.

DESCRIPTION OF THE RELATED ART

Currently, the TV and cinema market uses the scene extension technique,where a portion of the studio is physically mounted and another part iscreated by graphism.

The two major ways of implementing the expansion of scenarios today isthrough chrome-keying or through the use of projection headers.

The range of Scenarios with Chroma-keying is one of the most currentlyused techniques. In this case, the scenario is composed of floors andwalls of homogeneous color, generally green or blue, and may or may notcontain additional furniture. The process of replacing the image on thewall is done by computer program. Thus, all of the camera viewing in huewill be exchanged for an image, video or virtual graphism.

U.S. Pat. No. 6,091,579 describes a position detection device of avirtual studio object, which comprises providing an object with anemitting device and using the device in conjunction with a standardchroma-key background or background to provide the exact position of ATV camera or alone to provide a position for another object or person.

WO 197028654 describes a video system that creates a virtual shadow of afilmed object in front of a blue screen with a main video camera. Thevirtual shadow is created using A second camera located as A source ofvirtual light, this system uses the chroma-key color system.

The extent of scenarios with projection televisions consists of asimpler and easy-to-application technique, although quite restrictive.

In this case, a panel or telco is placed as an image, video or virtualscreen, through the presentor that is within the studio. This technique,however, limits the movements of the camera and, any wider movement mayshow the boundaries of the panel and impair the composition of theimage. In this technique, the use is restricted to stops or with minimalmovements.

The present invention provides a system that allows an integratedvirtual image to a real image, performed in real-time.

DRAWINGS

FIG. 1 shows the scenario extension using panels of LED.

FIG. 2 shows A perspective view of the carrot elements such as glassbottom and LED panels.

FIG. 3 shows a top view of the studio and the shape of the panels andcarrot elements.

FIG. 4 shows infographs and animated virtual character inserted into thevideo by the graphism servers.

FIG. 5 shows the physical layout of the studio, with the anchoringpoints of the panels.

FIG. 6 shows an embodiment of the asymmetric frustrum projectiontechnique.

FIG. 7 shows the distortions of an asymmetric frustrum.

DETAILED DESCRIPTION OF THE INVENTION

The present invention solves the aforementioned problems of studio sizeand moving the cameras offering A more complete and robust approach,delivering A final image where the television viewer is unable toperceive what is real or virtual and running all in real-time, ieallowable in live programs.

Thus, the presented solution enhances the perception of space in thestudio through the range concept of scenarios, making use of LEDTelevisions, glass bottom with view to a real environment, cameras withmotion tracking sensors and software responsible for controlling theimages displayed in the telco.

LED Big Displays

LED televisions are disposed laterally, perpendicular to the floor androtated relative to the background window forming an angle of 120degrees. This angle may vary, however, to be greater than 90 degrees.

The LED technology has been chosen for high resolution account by otherexisting technologies. This resolution is greater not only in the amountof pixels as in the intensity of the colors. There is an additionalimportance in the intensity of colors by accounting for the incidence ofsuch lighting on the real environment. This effect will contribute tothe perception of presentor insertion into the virtual environment.

Glass Background

The background is important to boost the effect of studio insertion in areal environment.

Near the glass bottom and hidden behind the LED side panels are twostatic cameras that capture images complementary to those viewed throughthe glass bottom. These complementary images are sent to servers thatdistort as the perspective of the studio camera and forwards them fordisplay on the side panels. When viewed through the camera of thestudio, the images on the panels extend the view offered by the glassbottom. However, if viewed from any other point of view other than thecamera, appear distorted and out of context.

Cameras with Motion Sensors

Two cameras with motion sensors are positioned within the studio. Theyhave the features of not only tracking movements, but also to promotechanges in the lens (zoom and focus). This data is captured and sent tothe graphics server that interprets them and thus deforms the imagescaptured by the static cameras near the bottom of the glass to bedisplayed in the televisions according to the perspective of the cameraspositioned within the studio.

Sensors and tracking systems are commonly known in the art. Software areused to generate and provide graphitization for projections on the LEDloops. An embodiment of the invention allows for the composition of afully virtual studio composed of LED Panels in the walls, roof andfloor.

The term “virtual Inserts” may be understood as any graphic computationinserted over the image such as: signs, characters, characters,characters, and graphics elements in general.

The physical studio is composed of an entire front wall of glass, andthe remaining opaque walls, behave in a transparent manner. The effectsought would be to recreate a 180 Degree (panoramic) studio and, forthis, the side walls should be glass, or invisible.

The physical studio can have any desired dimension, and the LED Panelsare then positioned on each side, rotated approximately 30 degrees, thepanel being the hypotenuse of the triangle formed therebetween, theglass and the wall, with the angle formed between the wall and thepanel.

The slope of the panels avoids aberrations in the image by virtue of thelight emitting angle of the LEDs, when viewed by the camera, otherwisemight suffer entirely on the total reflection effect and appear dark, inaddition to creating a hidden area for positioning between the panel andwall a static camera with a large angular (170 degree) lens bonded tothe glass to pick up the outside in process (called stiching).

The floor made on LED plates allows for the change of graphics, fordisplaying images or completing the current floor itself.

Application of filters and films is performed to arrive at luminancelevels in the glass compatible with equalizing the images generated inthe LED and images received through the front wall glass.

Even as an integral part of the vehicle's internal assembly, a sensorystick has been used for the purpose of camera tracking.

The process for the end effect is composed by the following steps:

capturing the images to complement the scene through 1 or 2 or morecameras with a large(s) angular lens;

the images of these cameras undergo a process called titching, which isto create a single image overlying the common zones between the 2cameras, having as a result a filmed panoramic image;

the panoramic image being sent to the graphical computing system, whichprojects it into a virtual plane that is spaced and scaled to overlieand cooperate with the actual image;

camera and lens tracking sensors are added so that, in real-time, thecameras used for transmission have their intrinsic and extrinsic data,being informed of the graphical computing system;

for each transmission camera, a graphical computing system is requiredreceiving all of the information mentioned above;

each graphical computation system calculates the projected distortion ofthe panoramic image, which is projected onto the LED panels while alwaysthe view point of the current camera. In this manner the image of theLEDs merges with the actual image of the window;

all video signals need to be adjusted by the TV systems so that theyhave the calculated delay, so that the videos are displayed withoutdelay and on the right order.

The graphical computing system is installed on servers that are locatedin the central art and receives the camera and lens tracking data by aconnection, such as ethernet.

The studio can be composed of two Side-LED Panels, optionally an LEDPanel on the floor and a glass window at the bottom. The side panels areperpendicular to the floor and rotated relative to the bottom windowforming an angle of 120 degrees (for the correct functioning, the angleshould be greater than 90 degrees).

Behind each side panel is one or more static cameras with one or morelarge angular lenses with a viewing angle near 180 degrees, pointing outof the studio through the glass window. The video captured by each ofthese cameras is inserted into a processor and sent to the correspondingside panel. This processor, in addition to the video of the rear-panelcamera, also receives the position and lens data from the main camera ofthe studio.

In an alternative embodiment, only one static camera can be used tocapture the external image.

In an alternative embodiment, the static camera may have one or morelarge angular lenses.

The projectile warping Process is performed by the asymmetric frustrumprojection technique, available and implemented in all programming andprogramming framework for real-time graphical computation already knownin the art.

The background distortion for asymmetric is also implemented byavailable software on the market, being represented by the functiondescribed below.

Since the system receives camera position data with respect to thestudio, i.e., identifying the camera on the studio, a virtual camera iscreated with the same characteristics and changes of lens, camera andposition occurring in real-time, these data are duplicated andsimultaneously feed two virtual systems simultaneously. The first system(system 1) processes all related to the distortions for the LED headers.The Second system (system 2) applies increased reality elements.

The LED panel has a processing time for displaying the images,therefore, once the system 2, responsible for the virtual inserts, needsto be delayed so that the system 1 protrudes on the LED the imagecomplementary to the glass before the system 2 Captures the studio imageto make the virtual inserts. The delay, referred to as aliasing, isvariable, lying in the 3-frame range in the worst case.

To enable the calculation of the projected distortion and otherresources, the servers in the central technique, where the systems areinstalled, are fed with tracking data from the camera and the lens, suchas via ethernet or any other compatible means, and with video, forexample via video cables. The system is divided into modules: theprojected distortion of the panoramic image is calculated by a part ofthe system that is a software available on the market; and the 3Delements inserted into the studio are generated by systems that will notbe described here.

The virtual window perception is due to the fact that temporal coherenceexists between the real image seen through the glass and the completionof the studio by the LED Panels. The effect as described generates aperception of a real scenario, much greater than the physical spaceavailable with imbedding interventions in fluid manner.

From the foregoing, it will be appreciated that numerous modificationsand variations may be made without departing from the true spirit andscope of the novel concepts of the present invention. It should beunderstood that no limitation with respect to the specific embodimentsillustrated is intended or should be inferred. The disclosure isintended to cover all such modifications as fall within the scope of theinvention.

1. An image acquisition and projection system, comprising LED panels,glass bottom with view to a real environment, cameras with motiontracking sensors, imaging caption static cameras and software operatingtogether.
 2. The image acquisition and projection system of claim 1,wherein the LED panels are disposed laterally with respect to the glassbottom, perpendicular to the floor and rotated relative to the bottomwindow.
 3. The image pickup and projection system of claim 2, whereinthe LED panels form an angle of greater than 90 degrees, preferably 120degrees.
 4. The image pickup and projection system of claim 1, whereinthe LED panels are rotated about 30 degrees.
 5. The image acquisitionand projection system of claim 1, further comprising one or more staticcameras behind the LED side panels.
 6. The image acquisition andprojection system of claim 5, wherein the static cameras send imagescomplementary to servers that distort as the perspective of the camerawith motion sensor and forward for display on the LED side Panels. 7.The image acquisition and projection system of claim 5, wherein thestatic cameras capture images that are deformed and displayed on the LEDtelevisions according to the perspective of the cameras with motionsensor positioned within the studio.
 8. The image acquisition andprojection system of claim 5, wherein the static cameras preferably havelarge(s) angular(s) lenses that capture images that complement theinternal scenario.
 9. The image acquisition and projection system ofclaim 5, wherein the images of the external static cameras will sufferstitching, creating a single image overlying the common zones betweenthe two cameras.
 10. The image acquisition and projection system ofclaim 5, wherein the static cameras create a panoramic image that issent to a graphical computing system that projects such an image into afar and scaled virtual plane to superimpose and interact with the realimage.
 11. The image acquisition and projection system of claim 1,further comprising two or more cameras with motion sensors positionedwithin a studio.
 12. The image acquisition and projection system ofclaim 11, wherein the cameras with motion sensors are capable oftracking movements and promoting changes in the lens as the movement.13. The image acquisition and projection system of claim 11, wherein themotion sensor cameras capture information that is sent to the graphismserver and interpreted.
 14. The image acquisition and projection systemof claim 11, wherein a graphical computing system receives tracking datafrom the cameras with motion sensor and lens by connection.
 15. Theimage acquisition and projection system of claim 11, wherein the motionsensor cameras further comprise camera and lens tracking sensors.
 16. Animage acquisition and projection system according to any one of theforegoing claims, wherein each graphical computing system calculates theprojected distortion of the panoramic image.
 17. The image acquisitionand projection system according to any of the foregoing claims, whereinthe video signals are set by TV systems having the calculated delay,wherein the videos are displayed without delay and in the correct order.18. Use of the image acquisition and projection system, as defined inany of the foregoing claims, characterized by being on a physicalstudio.
 19. The use of the image pickup and projection system of claim18, wherein the studio comprises an entire front wall of glass and theother opaque walls.
 20. The use of the image acquisition and projectionsystem of claim 18, wherein the entire front glass wall includesapplying filters and films that balance the images generated on the LEDwith the images received through the front wall glass.
 21. An imagecapturing and projection method using the system as defined in claims 1to 17, comprising the steps of: capturing the images complementary tothe real scene through one, two or more static cameras with largeangular lenses; sending the panoramic image to the graphical computingsystem and projecting it into a far and scaled virtual plane to overlieand cooperate with the real image; addition of camera and lens trackingsensors, the cameras used for real-time transmission having intrinsicand extrinsic data input to the graphical computing system; calculatingthe projected distortion of the panoramic image by the graphicalcomputing system; setting the video signals by the TV systems, whichcalculate the delay; display of the video without delay and in thecorrect order.